Phase Diagram of a Spin-Orbit Coupled Fermi Gases in a Bilayer Optical Lattice 
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We investigate the stability of helical superfluid phase in a spin-orbit coupled Fermi gas loaded in a 
bilayer optical lattice. The phase diagram of the system is constructed in the mean field framework. 
We investigate the topological properties of the superfluid phases by a nontrivial application of the 
Fermi surface topological invariant to our time-reversal invariant system with degeneracies on the 
Fermi surface. We find that there is a first-order phase boundary in the phase diagram of half 
filling case and the superfluid phases are all topological trivial. The superfluid phase is topological 
nontrivial when the filling fraction deviates from the half filling. In the topological nontrivial 
superfluid phase, a full pairing gap exists in the bulk and gapless helical Majorana edge states 
exist at the boundary. 

PACS numbers: 03.75.Ss, 03.65.Vf, 05.30.Fk. 



INTRODUCTION 



Recently, the topological properties have been investi- 
gated extensively in condensed matter systems such as 
topological insulators (TIs) [1—5], topological superffu- 
ids (TSFs) /superconductors (TSCs) [6-12]. TSFs/TSCs 
have a full pairing gap in the bulk and gapless Majo- 
rana edge states at the boundary. These new quantum 
phases are described by topological order [13] instead of 
the traditional Landau symmetry breaking theory. The 
gapless Majorana edge states are protected by the sym- 
metry of bulk for the bulk-edge correspondence. For the 
application, the gapless Majorana edge states have sig- 
nificant application in topological quantum computation 
and attract considerable attention [14, 15]. In two dimen- 
sions(2D), there are two classes of TSFs/TSCs, namely 
the chiral supcrfluids/superconductors and helical supcr- 
fluids/supconductors. Chiral supcrfluids/supconductors 
break the time-reversal symmetry and have chiral Majo- 
rana edge states at the boundary [6, 7, 10]. Whereas 
the helical superfluids/supconductors are time-reversal 
invariant (TRI) and have gapless helical Majorana edge 
states at the boundary [16-18]. They are classified by an 
integer and a Z2 invariant respectively [9] . The helical su- 
perfluid phase can be stabilized in a two-band spin-orbit 
coupled system [17] or a bilayer system[18] with SOC and 
repulsive interlayer interaction. 

In ultracold fermonic system, TIs/TSFs are proposed 
to be realized by utilizing spin-orbit coupling (SOC). 
Most recently, the effective SOC has been successfully 
realized both in ultracold bosonic [19, 20] and fermonic 
[21, 22] systems and has generated a considerable amount 
of theoretical interests [23-35] . With remarkable tunabil- 
ity and clean environment [36-38] , ultracold Fermi gases 
provide an ideal platform for investigating the interest- 
ing topological properties. For a spin-polarized Fermi 



gas, the chiral superfluid phases can be stabilized in the 
presence of spin-orbit coupling and exhibit many new in- 
teresting physics [7, 8, 39-41]. However, the study of 
the helical superfluid phase are destitute in ultracold fer- 
monic system. In addition, a bilayer fermionic system 
exhibits many new interesting phenomena [43-45] com- 
pared to the singlclaycr case. With a SOC, the bilayer 
fermionic system can stabilize the helical superfluid phase 
without breaking the time-reversal symmetry. 



In this paper, we investigate a spin-orbit coupled Fermi 
gas loaded in a bilayer optical lattice by using a BCS-type 
mean field theory at zero temperature. The interlayer 
hopping is included instead of the interlayer interaction. 
The ground state is self-consistently determined by min- 
imizing the thermodynamic potential. Due to the degen- 
eracies on the Fermi surfaces, we investigate the topo- 
logical properties of the superfluid phases by a nontriv- 
ial application of the Fermi surface topological invariant 
(FSTI). Our main results are the followings. First, the 
superfluid phases are all topological trivial in the phase 
diagram of half filling case while there is a first-order 
phase boundary. Second, the superfluid phase is topo- 
logical nontrivial when the filling fraction deviate from 
the half filling. Without breaking the time-reversal sym- 
metry (TRS), the topological nontrivial superfluid phase 
is helical superfluid phase. This state has a full pairing 
gap in the bulk and gapless helical Majorana edge states 
countcrpropagating at the boundary. In the phase dia- 
gram of filling fraction, there are two helical superfluid 
phases which are separated by a gapless boundary. There 
is no difference between the two helical superfluid phases 
in topology. Lastly, the effect of the trapping potential is 
discussed under local density approximation. Our work 
provides a possible route to realize the helical superfluid 
phase in ultracold fermionic system in the future. 
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FORMALISM OF THE SYSTEM 

We consider a system of bilayer fermi gas of isotropic 
spin-orbit coupling. In the mean field framework, the 
system's Hamiltonian can be given as the following: 



H = H + H, 



so 



(1) 



where Hq is kinetic term, Hgo is isotropic SOC term and 
H s is s-wave pairing term. The following are their detail 
expression 



H = ^ ~ f^i,j + th8i,jT x )lpj, (2) 

<i,i> 

H S o = - E Wii™ X e)zT z ll>i+e + h.c], (3) 
z,e— x.y 

h s = - J2( A 4A Ab lA + h - c -)- ( 4 ) 

i 

Here,^ = (<Zi,t, a,,j., &i,f, &i,j.) T with a(a t ) and 6(6 t ) de- 
noting the fermion annihilation (creation) operators for 
A and -B layers, respectively. The Pauli matrices a act 
on the spin and r acts on the layer, is the strength of 
interlayer hopping. In the above forms, we have assumed 
that there is a 7r phase shift between the SOCs of the two 
layers, A is the strength of the SOC. We have also as- 
sumed = — A B = A [42] with A^ = —U < a^a^ > 
and Ab = —U < b^b^ >, where U is the strength of 
the attractive on-site interaction. We only consider the 
intralayer on-site interaction and the interlayer hopping 
for simplicity. 

Therefore, by introducing tp^ = (a k , 6 k , a_k, &-k) with 
Fourier transformations a; = ^ k exp(ik • r^a^/y/N and 
bi = ^ k cxp(ik • ri)&k/ VN (N is the number of the sites 
of the lattices), the Hamiltonian is given by 



(5) 



with 
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with E = V^( A2 +^k)+^klSk| 2 - The topological 
phase transition only occurs at the closing points of 
the gap, that is to say, the topological invariants may 
change only when the gap closes. In the presence of pair- 
ing gap, the excitation spectrum is gapless only when 
|gk| =0 and th = + A 2 which can be simplified as 

t h = + A2 With k c G {(0, 0), [(0, 7T), (7T, 0)], (7T, 7T)}. 

The thermodynamic potential is 17 = — Trln[e _/3H ] 
with /3 _1 = ksT. At zero temperature, the thermody- 
namic potential is 



= ^ + £(6k-£„(k)). 



U 



(8) 



The pairing gap and the chemical potential should be 
determined self-consistcntly by minimizing the thermo- 
dynamic potential. The gap and number equations are 
given as 
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(9) 
.(10) 



By self-consistcntly solving above equations, we have 
found the phase diagram of system. The topological in- 
variant properties will be discussed in the next section. 



TOPOLOGICAL PROPERTIES 

Rigorous topological invariants of interacting insula- 
tors and superfluids/supcrconductors defined in terms of 
'topological Hamiltonian', which is the inverse of zero 
frequency Green's function h t = — G _1 (0,k) , have been 
proposed in Ref.[5, 11, 12] recently. In the mean-field 
approximation and weakly pairing limit, we can also use 
the FSTI given by Qi. et al in Ref.[16], which takes the 
following form 



V 

where, 6k = —2t(cosk x + cosk y ) — \i and g k = 
2A(sinfc l/ , — sinfc x ). Here, the Hamiltonian of two dimen- 
sion Bogoliubov-de Gennes (BdG) possess the partical- 
hole symmetry (PHS) and TRS. Therefore, the classi- 
fication of the Hamiltonian is Z 2 class. Diagonalizing 
the Hamiltonian, the excitation spectrum E± (k) of the 
quasiparticles are 
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E ± (k) 
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gk 



t% ± 2E . 



(7) 



\lhere is the number of TRI points enclosed by the 
ith Fermi surface, sgn(5i) is the sign of pairing gap on 
the ith Fermi surface of single particle Hamiltonian hk 
with <5j,k =< i, k|TA^|i, k >, where \i, k > are the eigen- 
vectors of /ik and A = iAa y T z . The 2D TRI supcrfluid 
phase is topological trivial for N 2 o = 1 and nontrivial for 
N 2 d = — 1. The FSTI shows that the topological proper- 
ties of a superfluid phase is completely determined by the 
Fermi surface properties in weakly pairing limit. For our 
system, the pairing gap can be larger than the strength 
of the hopping(i), thus the FSTI is not well defined in 
these regions. Even though the FSTI can only be applied 
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FIG. 1: Fermi surfaces of Ea^k) in the first Brillouin Zone 
with h — for th = 2t, A = t with (a) /j, = — t, (b) fi — t, 
(c) jj, — —At and (d) /i = At. The number of TRI points 
enclosed by Fermi surfaces is two for (a), (b) and one for 
(c),(d) respectively. 

to the narrow supcrfiuid region near the boundaries of 
the normal and supcrfiuid phases, the information is ad- 
equate for determining the topological invariants of the 
supcrfiuid phases in the phase diagrams because we ex- 
pect that topological properties should be the same for 
the same phase. 

The eigenvalues of the single particle Hamiltonian are 
Ei(k) = £k ± yif + | gk | 2 which are with twofold de- 
generacy. For a system with degenerate Fermi surfaces, 
the above FSTI cannot work and some perturbations 
should be introduced to lift the degeneracy while pre- 
serving the PHS and TRI symmetry of the system. To 
lift the degeneracy, we can introduce an imbalance in 
chemical potential h = (fis — Ma)/2 of the two layers 
with fj, = (fjLA + Mb)/2, then the single particle Hamil- 
tonian changes into /i k = + hr z with Pauli matrix 
t z act on the layer part, hence the four eigenvalues of 
the imbalanced Hamiltonian Ei(k) (-Ei(k) < £7 2 (k) < 
i?3(k) < £4(k)) have no degeneracy and the FSTI can 
work well. The sign of the pairing gap on the Fermi sur- 
faces is sgn(5i j3 ) = —1, sgn(<5 2 ,4) = 1 for |gk| > h and 
sgn(<5i,2) = — 1, sgn(53 j 4) = 1 for |gk| < h (More details 
are included in the APPENDIX). 

For |gk| < h, the imbalanced Hamiltonian cannot adi- 
abatically changes into the balanced case by taking the 
limit h — > 0, this indicates that there exists topological 
phase transitions as h increasing. In this work, we only 
concentrate on the balanced case and leave the imbalance 
case for future discussion. 

For |gk| > h, the imbalanced Hamiltonian can be adi- 
abatically changed into the balanced case by taking the 
limit h — > without changing any topological proper- 
ties, thus the sign of the pairing gap on the Fermi sur- 
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FIG. 2: (left)The phase diagram as a function of U and th 
with A = t and n — 1 . (right)The phase diagram as a function 
of filling fraction n and the interlayer hopping th with A = t 
and U = &t. The dotted curve represents the gapless bound- 
ary and the dashed-dotted curve represents first-order phase 
boundary. The dashed curve represents topological phase 
boundaries. The superfiuid phases are all topological non- 
trivial for n = 1. There are two topological superfiuid regions 
separated by a dotted curve when the filling fraction deviates 
from the half filling. 

faces for the balanced case is the same as the imbalanced 
case. Such that the FSTI only depends on the number 
of the TRI points enclosed by the Fermi surfaces of the 
Ei^(k). Fig.l shows the Fermi surfaces of Ei^(k) for 
various cases. The number of the TRI points enclosed by 
the Fermi surfaces is 2 for (a), (b) and 1 for (c), (d). Ac- 
cording to analysis of the relationship between E\ (k) and 
i?3(k), we have m\ + amounts to for th > 4t + |ju| 
, 2 for th < 4t - \fj,\ and 1 for At - < t h < 4t + 
Therefore, the superfiuid phases are topological nontriv- 
ial when the interlayer hopping satisfies the topological 
condition At — \fi\ < th, < At + \fi\ in the weakly pairing 
limit. 

TOPOLOGICAL SUPERFLUID PHASE 

The topological phase transition only occurs at the 
gapless points with the changing of the topological invari- 
ants. Thus, the gapless conditions show the significance 
of determining the topological invariants of the superfiuid 
phases in the phase diagram. Since the supcrfiuid phases 
are topological trivial in the absence of interlayer hopping 
(th = 0), the topological invariant of the superfiuid phase 
near the boundaries between the normal and supcrfiuid 
phases can be determined by the FSTI. Therefore, the 
topological invariants of all the superfiuid phases in the 
phase diagram can be determined. 

First, we show the phase diagram as a function of on- 
site interaction (U) and interlayer hopping (th) for half 
filling case in Fig. 2 (a). The dotted curve represents the 
gapless boundary and the dashed-dotted curve represents 
first-order phase boundary along which the thermody- 
namic potential has two degenerate minima. By analyz- 
ing the relationship between [i and th along the boundary 
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FIG. 3: The spectrum of Hamiltonian 1 in a strip geometry 
for (a) fi = — 4t, th = St and (b) /Lt = 0, th = 3t (c) /i = 4t, 
t h = 5t (d) /i = -4i, t h = 5t with A = t and A = t. 



between the normal and the supcrfiuid phases, we have 
N 2 d = 1 for all the superfluid phase which indicates that 
there is no topological nontrivial superfluid phase for the 
half filling case. 

Second, in order to find the topological nontrivial su- 
perfluid phase, we investigate the case as deviating from 
the half filling. As the system has particle-hole symmetry 
at half filling, the phase diagram has an axial symmetry. 
Fig. 2(b) shows the phase diagram as a function of filling 
fraction (n) and the interlayer hopping (th). The dashed 
and the dotted curves represent the gapless boundaries 
along which the gapless points are (0,0) ((ir,ir)) and 
[(0,7r), (7t,0)] for fx < (fj, > 0) respectively. Therefore, 
the chemical potential and the interlayer hopping satisfy 
the topological condition At — < th < At + at the 
supcrfiuid side of the normal-supcrfluid boundary which 
is above the dashed curve. Without the interlayer hop- 
ping, the system can be considered as two independent 
SO coupled Fermi gases in which the supcrfiuid phases 
are all topological trivial. As the interlayer hopping in- 
creases, there exists a topological phase transition when 
the filling fraction deviates from the half filling. As shown 
in Fig.2(b), there are two TSF phases. The two TSF 
phases are helical superfluid phases and have no differ- 
ence in topology. The only difference between the two 
helical superfluid phases is that the TRI points enclosed 
by the- Fermi surfaces are different in the weakly pairing 
limit. 

As mentioned before, the TSF phases are also charac- 
terized by the gapless boundary modes. For a 2D TRI 
TSF phase, the boundary modes are gapless helical Ma- 
jorana edge states, where time-reversal partners counter- 
propagate. Accordingly, each TRI topological defect of 
the TSFs carries a Kramers pair of Majorana fcrmions. 
To investigate the helical Majorana edge states, we cal- 



culate the spectrum of the Hamiltonian (1) in a strip 
geometry where the edges are along the y direction. Due 
to breaking of translation symmetry along the y direc- 
tion, k y is not a good quantum number while k x is a 
good quantum number, and then we perform a Fourier 
transformation along the a; direction only. Fig. 3 show 
the spectrum of the Hamiltonian as a function of k x for 
various [i and th with A = t and A = t. Fig. 3(a) and (d) 
are the cases of the < n < 1 while the Fig. 3(c) is the 
case 1 < n < 2. The three phases have a full gap in the 
bulk and gapless helical Majorana edge states counter- 
propagating at the boundary. Fig. 3(b) is the half filling 
case and there are two pairs of gapless Majorana edge 
states at the boundary, which indicates that the phase is 
topological trivial. 

At last, we consider the effects of a weakly harmonic 
trapping potential under LDA. In the presence of har- 
monic trapping potential, the global chemical potential 
(/ii = fi — muj 2 r?) and the pairing gap are the functions of 
spatial coordinates. The topological invariants of the su- 
perfluid phases depend on the spatial coordinates. There- 
fore, the helical superfluid phase and topological triv- 
ial phases (normal phase and topological trivial super- 
fluid phase) can coexist in the trapped region and a shell 
structure of topological phase separation emerges[39-41]. 
The helical Majorana edge states, which can be called 
helical Majorana liquid, exists at the interfaces of the 
helical superfluid phase and topological trivial phases. 
The position of the helical Majorana liquid is controlled 
by the parameters of the system. The existence of the 
helical Majorana liquid will lead to nontrivial physical 
consequences [18]. In addition, we only consider the in- 
terlayer hopping and the intralayer on-site attractive in- 
teraction for simplicity. When the interlayer interaction 
is taken into account, the phase diagram will be enriched. 
Furthermore, the helical superfluid phase with unconven- 
tional pairing like the d x 2_ y 2 might be stabilized when 
the on-site interaction is replaced by the nearest-neighbor 
interaction. 



CONCLUSIONS 

To conclude, we have constructed the phase diagram 
for a bilayer spin-orbit coupled Fermi gas loaded in a 
square optical lattice. We have found that the TRI su- 
pcrfiuid phases are topological nontrivial when the filling 
fraction deviates from the half filling. There are helical 
edge states at the boundary of the TRI topological su- 
perfluid phase. In addition, we also have discussed the 
effects of a harmonic trapping potential. There is a shell 
structure of topological phase separation phenomena in 
the trapped region. We expect our work can provide a 
possible route to realize the helical superfluid phase by 
ultracold fcrmionic system in the future. 
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APPENDIX: DERIVATION OF THE FSTI 

The imbalanced single particle Hamiltonian is 
6c + h + g k • a t h 



h' k = 



(12) 



The four nondegenerate eigenvalues and the corre- 
sponding eigenvectors of h' k are 



^(k) = &±^ + (|gk|±/0 2 , 



/ / x , ■ ;n lg k l+W^ + ag k |+'') 2 \ 

\.J fi J K / tu 
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(13) 
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t h 
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(17) 



/ 



with Si(k) < E 2 (k) < E 3 (k) < £ 4 (k). 

The time-reversal matrix and the pairing gap matrix 
are 



T 



lOy 

ia v 



iA<jy 






-iA<7„ 



(18) 



By inserting the pairing gap and the eigenvectors into 
5i k, we have 



4(| gk | +fe) ^±W g + (lg k l^) 2 : 

4(|gk| - ft) 
4(|g k |-fe) ISkhfe -V|+(l^l- fc ) 2 



Igkl-^+V^ + dgkh/i) 2 



4(|gk| + h) 



Igkl + fe+ytl + dgkl+Z') 2 



= 1 
2 
3 
4 



(19) 



The sign of the pairing gap on the Fermi surfaces is 
sgn(t5i, 3 ) = -1, sgn((5 2 ,4) = 1 for |g k | > h and sgn(5i j2 ) = 
-1, sgn(5 3)4 ) = 1 for |g k | < h. 
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